A computer network is a geographically distributed collection of interconnected communication links and segments for transporting data between nodes, such as computers. Many types of network segments are available, with the types ranging from local area networks (LAN) to wide area networks (WAN). For example, the LAN may typically connect personal computers and workstations over dedicated, private communications links, whereas the WAN may connect large numbers of nodes over long-distance communications links, such as common carrier telephone lines. The Internet is an example of a WAN that connects disparate networks throughout the world, providing global communication between nodes on various networks. The nodes typically communicate over the network by exchanging discrete frames or packets of data according to predefined protocols. In this context, a protocol consists of a set of rules defining how the nodes interact with each other.
Computer networks may be further interconnected by an intermediate node, such as a switch or router, having a plurality of ports that may be coupled to the networks. To interconnect dispersed computer networks and/or provide Internet connectivity, many organizations rely on the infrastructure and facilities of Internet Service Providers (ISPs). ISPs typically own one or more backbone networks that are configured to provide high-speed connection to the Internet. To interconnect private networks that are geographically diverse, an organization may subscribe to one or more ISPs and couple each of its private networks to the ISP's equipment. Here, the router may be utilized to interconnect a plurality of private networks or subscribers to an IP “backbone” network. Routers typically operate at the network layer of a communications protocol stack, such as the internetwork layer of the Transmission Control Protocol/Internet Protocol (TCP/IP) communications architecture.
Simple networks may be constructed using general-purpose routers interconnected by links owned or leased by ISPs. As networks become more complex with greater numbers of elements, additional structure may be required. In a complex network, structure can be imposed on routers by assigning specific jobs to particular routers. A common approach for ISP networks is to divide assignments among access routers and backbone routers. An access router provides individual subscribers access to the network by way of large numbers of relatively low-speed ports connected to the subscribers. Backbone routers, on the other hand, provide transports to Internet backbones and are configured to provide high forwarding rates on fast interfaces. ISPs may impose further physical structure on their networks by organizing them into points of presence (POP). An ISP network usually consists of a number of POPs, each of which comprises a physical location wherein a set of access and backbone routers is located.
As Internet traffic increases, the demand for access routers to handle increased density and backbone routers to handle greater throughput becomes more important. In this context, increased density denotes a greater number of subscriber ports that can be terminated on a single router. Such requirements can be met most efficiently with platforms designed for specific applications. An example of such a specifically designed platform is an aggregation router. The aggregation router is an access router configured to provide high quality of service and guaranteed bandwidth for both data and voice traffic destined for the Internet. The aggregation router also provides a high degree of security for such traffic. These functions are considered “high-touch” features that necessitate substantial processing of the traffic by the router. Notably, the aggregation router is configured to accommodate increased density by aggregating a large number of leased lines from ISP subscribers onto a few trunk lines coupled to an Internet backbone.
When designing an intermediate node, such as an aggregation router, it may be necessary to place an external device controlling an interrupt source behind a shared bus or other path having high latency on data transfers. However, this architecture may lead to long delays in accessing that external device when acknowledging the interrupt. As an example, consider an aggregation router comprising a central processing unit (CPU) and CPU memory coupled to an external bus, such as a conventional peripheral computer interconnect (PCI) bus, via a system controller. An external device, such as a direct memory access (DMA) controller, is coupled to the PCI bus via a PCI bridge. Since the PCI bus is shared among the DMA controller, the PCI bridge and the system controller, each device must arbitrate for access to the bus prior to transferring data over the bus. As a result, the PCI bus is a slow, high latency path.
The DMA controller performs DMA operations to and from the CPU memory over the PCI bus. That is, the DMA controller moves data, such as packets, over the PCI bus and bridge, through the system controller and to memory for processing by the CPU. In addition, the controller moves packets from the CPU memory through the system controller over the PCI bus and bridge to destinations within the router. The DMA controller notifies the CPU of completion of these DMA operations through the use of an interrupt signal. Although the CPU is notified of an interrupt, it requires further information to determine the type and source of the interrupt.
The CPU typically obtains such further information by retrieving the contents of a register within the DMA controller. For example, the CPU may retrieve the contents of an interrupt status register (ISR) which stores information (e.g., a status bit) identifying the type and source of the interrupt. Each source that asserts an interrupt has a corresponding asserted status bit within the ISR. Depending upon the type of interrupt and the particular source(s) generating the interrupt, appropriate interrupt handler code is invoked by the CPU to service the interrupt. The handler may then examine a data structure in the CPU memory that is shared between the CPU and DMA controller in order to access the data associated with the interrupt.
The shared data structure typically has a plurality of control blocks that point to (references) buffers in the memory where data, e.g., packets, associated with the interrupt reside. Each control block includes an ownership bit denoting ownership of the control structure by the DMA controller or the CPU. When it has a packet to move into CPU memory, the DMA controller examines the state of the ownership bit to determine whether it is the owner of that control block. If it is, the DMA controller moves the data packet into the buffer specified by the control block. After the data transfer is completed, the DMA controller changes the state of the ownership bit to reflect ownership of the control block by the CPU. The DMA controller then issues the interrupt signal to the CPU which, subsequently, enters the interrupt handler to search the shared data structure for a control block indicating ownership by the CPU.
Specifically, the DMA controller performs a write operation over the PCI bus to move the packet into a CPU memory buffer referenced by the control block and then performs another write operation to change the state of the ownership bit for that control block. After issuing the change of ownership operation over the PCI bus, the DMA controller generates and asserts the interrupt signal. Yet when the CPU/handler accesses its memory in response to the interrupt, it is possible that the data packet has not yet been written (stored) at the appropriate memory location. Moreover, the data packet may be stored at the appropriate memory location, but the ownership bit may not yet have been updated by the change of ownership operation because it is “stalled” within buffers or queues of the devices coupled to the PCI bus.
When the CPU issues a read operation to retrieve the contents of the ISR within the DMA controller to determine the type of interrupt, that read operation also functions to ensure that any pending write operations from the DMA controller to the CPU memory have been “flushed” from those queues. That is, the read operation that informs the CPU as to the type of interrupt generated by the DMA controller also ensures that the data packet and ownership bit transfers have been completed to the CPU memory. In addition, the read operation acknowledges and clears the interrupt at the DMA controller. Thus, the CPU acknowledges and clears the interrupt by retrieving the contents of the ISR in the DMA controller. However, the read operation generally takes a long time to complete because devices coupled to the PCI bus, such as the PCI bridge, need to arbitrate for the shared bus. The present invention is directed to reducing the time it takes to acknowledge and clear the interrupt and, in essence, reducing the latency caused by a read operation over a slow bus of the router.